The phrase "chilly nights" refers to cold or cool weather during the evening time. It describes a situation where temperatures drop significantly after sunset, making it uncomfortable for people who are not properly dressed or prepared for such conditions.